Antenna Season Report Notebook¶

Josh Dillon, Last Revised January 2022

This notebook examines an individual antenna's performance over a whole season. This notebook parses information from each nightly rtp_summarynotebook (as saved to .csvs) and builds a table describing antenna performance. It also reproduces per-antenna plots from each auto_metrics notebook pertinent to the specific antenna.

In [1]:
import os
from IPython.display import display, HTML
display(HTML("<style>.container { width:100% !important; }</style>"))
In [2]:
# If you want to run this notebook locally, copy the output of the next cell into the next line of this cell.
# antenna = "004"
# csv_folder = '/lustre/aoc/projects/hera/H5C/H5C_Notebooks/_rtp_summary_'
# auto_metrics_folder = '/lustre/aoc/projects/hera/H5C/H5C_Notebooks/auto_metrics_inspect'
# os.environ["ANTENNA"] = antenna
# os.environ["CSV_FOLDER"] = csv_folder
# os.environ["AUTO_METRICS_FOLDER"] = auto_metrics_folder
In [3]:
# Use environment variables to figure out path to the csvs and auto_metrics
antenna = str(int(os.environ["ANTENNA"]))
csv_folder = os.environ["CSV_FOLDER"]
auto_metrics_folder = os.environ["AUTO_METRICS_FOLDER"]
print(f'antenna = "{antenna}"')
print(f'csv_folder = "{csv_folder}"')
print(f'auto_metrics_folder = "{auto_metrics_folder}"')
antenna = "113"
csv_folder = "/home/obs/src/H6C_Notebooks/_rtp_summary_"
auto_metrics_folder = "/home/obs/src/H6C_Notebooks/auto_metrics_inspect"
In [4]:
display(HTML(f'<h1 style=font-size:50px><u>Antenna {antenna} Report</u><p></p></h1>'))

Antenna 113 Report

In [5]:
import numpy as np
import pandas as pd
pd.set_option('display.max_rows', 1000)
import glob
import re
from hera_notebook_templates.utils import status_colors, Antenna
In [6]:
# load csvs and auto_metrics htmls in reverse chronological order
csvs = sorted(glob.glob(os.path.join(csv_folder, 'rtp_summary_table*.csv')))[::-1]
print(f'Found {len(csvs)} csvs in {csv_folder}')
auto_metric_htmls = sorted(glob.glob(auto_metrics_folder + '/auto_metrics_inspect_*.html'))[::-1]
print(f'Found {len(auto_metric_htmls)} auto_metrics notebooks in {auto_metrics_folder}')
Found 74 csvs in /home/obs/src/H6C_Notebooks/_rtp_summary_
Found 72 auto_metrics notebooks in /home/obs/src/H6C_Notebooks/auto_metrics_inspect
In [7]:
# Per-season options
mean_round_modz_cut = 4
dead_cut = 0.4
crossed_cut = 0.0

def jd_to_summary_url(jd):
    return f'https://htmlpreview.github.io/?https://github.com/HERA-Team/H6C_Notebooks/blob/main/_rtp_summary_/rtp_summary_{jd}.html'

def jd_to_auto_metrics_url(jd):
    return f'https://htmlpreview.github.io/?https://github.com/HERA-Team/H6C_Notebooks/blob/main/auto_metrics_inspect/auto_metrics_inspect_{jd}.html'

Load relevant info from summary CSVs¶

In [8]:
this_antenna = None
jds = []

# parse information about antennas and nodes
for csv in csvs:
    df = pd.read_csv(csv)
    for n in range(len(df)):
        # Add this day to the antenna
        row = df.loc[n]
        if isinstance(row['Ant'], str) and '<a href' in row['Ant']:
            antnum = int(row['Ant'].split('</a>')[0].split('>')[-1]) # it's a link, extract antnum
        else:
            antnum = int(row['Ant'])
        if antnum != int(antenna):
            continue
        
        if np.issubdtype(type(row['Node']), np.integer):
            row['Node'] = str(row['Node'])
        if type(row['Node']) == str and row['Node'].isnumeric():
            row['Node'] = 'N' + ('0' if len(row['Node']) == 1 else '') + row['Node']
            
        if this_antenna is None:
            this_antenna = Antenna(row['Ant'], row['Node'])
        jd = [int(s) for s in re.split('_|\.', csv) if s.isdigit()][-1]
        jds.append(jd)
        this_antenna.add_day(jd, row)
        break
In [9]:
# build dataframe
to_show = {'JDs': [f'<a href="{jd_to_summary_url(jd)}" target="_blank">{jd}</a>' for jd in jds]}
to_show['A Priori Status'] = [this_antenna.statuses[jd] for jd in jds]

df = pd.DataFrame(to_show)

# create bar chart columns for flagging percentages:
bar_cols = {}
bar_cols['Auto Metrics Flags'] = [this_antenna.auto_flags[jd] for jd in jds]
bar_cols[f'Dead Fraction in Ant Metrics (Jee)'] = [this_antenna.dead_flags_Jee[jd] for jd in jds]
bar_cols[f'Dead Fraction in Ant Metrics (Jnn)'] = [this_antenna.dead_flags_Jnn[jd] for jd in jds]
bar_cols['Crossed Fraction in Ant Metrics'] = [this_antenna.crossed_flags[jd] for jd in jds]
bar_cols['Flag Fraction Before Redcal'] = [this_antenna.flags_before_redcal[jd] for jd in jds]
bar_cols['Flagged By Redcal chi^2 Fraction'] = [this_antenna.redcal_flags[jd] for jd in jds]
for col in bar_cols:
    df[col] = bar_cols[col]

z_score_cols = {}
z_score_cols['ee Shape Modified Z-Score'] = [this_antenna.ee_shape_zs[jd] for jd in jds]
z_score_cols['nn Shape Modified Z-Score'] = [this_antenna.nn_shape_zs[jd] for jd in jds]
z_score_cols['ee Power Modified Z-Score'] = [this_antenna.ee_power_zs[jd] for jd in jds]
z_score_cols['nn Power Modified Z-Score'] = [this_antenna.nn_power_zs[jd] for jd in jds]
z_score_cols['ee Temporal Variability Modified Z-Score'] = [this_antenna.ee_temp_var_zs[jd] for jd in jds]
z_score_cols['nn Temporal Variability Modified Z-Score'] = [this_antenna.nn_temp_var_zs[jd] for jd in jds]
z_score_cols['ee Temporal Discontinuties Modified Z-Score'] = [this_antenna.ee_temp_discon_zs[jd] for jd in jds]
z_score_cols['nn Temporal Discontinuties Modified Z-Score'] = [this_antenna.nn_temp_discon_zs[jd] for jd in jds]
for col in z_score_cols:
    df[col] = z_score_cols[col]

ant_metrics_cols = {}
ant_metrics_cols['Average Dead Ant Metric (Jee)'] = [this_antenna.Jee_dead_metrics[jd] for jd in jds]
ant_metrics_cols['Average Dead Ant Metric (Jnn)'] = [this_antenna.Jnn_dead_metrics[jd] for jd in jds]
ant_metrics_cols['Average Crossed Ant Metric'] = [this_antenna.crossed_metrics[jd] for jd in jds]
for col in ant_metrics_cols:
    df[col] = ant_metrics_cols[col]

redcal_cols = {}
redcal_cols['Median chi^2 Per Antenna (Jee)'] = [this_antenna.Jee_chisqs[jd] for jd in jds]
redcal_cols['Median chi^2 Per Antenna (Jnn)'] = [this_antenna.Jnn_chisqs[jd] for jd in jds]   
for col in redcal_cols:
    df[col] = redcal_cols[col]

# style dataframe
table = df.style.hide_index()\
          .applymap(lambda val: f'background-color: {status_colors[val]}' if val in status_colors else '', subset=['A Priori Status']) \
          .background_gradient(cmap='viridis', vmax=mean_round_modz_cut * 3, vmin=0, axis=None, subset=list(z_score_cols.keys())) \
          .background_gradient(cmap='bwr_r', vmin=dead_cut-.25, vmax=dead_cut+.25, axis=0, subset=list([col for col in ant_metrics_cols if 'dead' in col.lower()])) \
          .background_gradient(cmap='bwr_r', vmin=crossed_cut-.25, vmax=crossed_cut+.25, axis=0, subset=list([col for col in ant_metrics_cols if 'crossed' in col.lower()])) \
          .background_gradient(cmap='plasma', vmax=4, vmin=1, axis=None, subset=list(redcal_cols.keys())) \
          .applymap(lambda val: 'font-weight: bold' if val < dead_cut else '', subset=list([col for col in ant_metrics_cols if 'dead' in col.lower()])) \
          .applymap(lambda val: 'font-weight: bold' if val < crossed_cut else '', subset=list([col for col in ant_metrics_cols if 'crossed' in col.lower()])) \
          .applymap(lambda val: 'font-weight: bold' if val > mean_round_modz_cut else '', subset=list(z_score_cols.keys())) \
          .applymap(lambda val: 'color: red' if val > mean_round_modz_cut else '', subset=list(z_score_cols.keys())) \
          .bar(subset=list(bar_cols.keys()), vmin=0, vmax=1) \
          .format({col: '{:,.4f}'.format for col in z_score_cols}) \
          .format({col: '{:,.4f}'.format for col in ant_metrics_cols}) \
          .format('{:,.2%}', na_rep='-', subset=list(bar_cols.keys())) \
          .set_table_styles([dict(selector="th",props=[('max-width', f'70pt')])]) 

Table 1: Per-Night RTP Summary Info For This Atenna¶

This table reproduces each night's row for this antenna from the RTP Summary notebooks. For more info on the columns, see those notebooks, linked in the JD column.

In [10]:
display(HTML(f'<h2>Antenna {antenna}, Node {this_antenna.node}:</h2>'))
HTML(table.render(render_links=True, escape=False))

Antenna 113, Node N11:

Out[10]:
JDs A Priori Status Auto Metrics Flags Dead Fraction in Ant Metrics (Jee) Dead Fraction in Ant Metrics (Jnn) Crossed Fraction in Ant Metrics Flag Fraction Before Redcal Flagged By Redcal chi^2 Fraction ee Shape Modified Z-Score nn Shape Modified Z-Score ee Power Modified Z-Score nn Power Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Discontinuties Modified Z-Score nn Temporal Discontinuties Modified Z-Score Average Dead Ant Metric (Jee) Average Dead Ant Metric (Jnn) Average Crossed Ant Metric Median chi^2 Per Antenna (Jee) Median chi^2 Per Antenna (Jnn)
2459890 not_connected 100.00% 100.00% 100.00% 0.00% - - 12.921914 13.611979 4.799823 5.612753 8.080866 9.069373 0.906066 -0.175549 0.0341 0.0308 0.0020 nan nan
2459889 not_connected 100.00% 100.00% 100.00% 0.00% - - 14.375223 15.368499 3.558296 4.620022 10.966863 12.881762 3.052664 1.297265 0.0364 0.0310 0.0033 nan nan
2459888 not_connected 100.00% 100.00% 100.00% 0.00% - - 12.042096 12.815486 4.643629 5.465487 9.848987 11.787880 2.392165 2.057200 0.0362 0.0308 0.0033 nan nan
2459887 not_connected 100.00% 100.00% 100.00% 0.00% - - 12.895705 13.696026 4.803014 5.371364 7.380940 10.296628 1.480593 0.754102 0.0365 0.0309 0.0033 nan nan
2459886 not_connected 100.00% 100.00% 100.00% 0.00% - - 17.993098 18.618152 4.210142 4.813103 14.553657 14.978108 3.927069 3.236074 0.0396 0.0312 0.0051 nan nan
2459885 not_connected 100.00% 100.00% 100.00% 0.00% - - 21.950170 22.441739 23.551904 26.014454 14.145887 19.401089 16.917254 11.542472 0.0355 0.0308 0.0026 nan nan
2459884 not_connected 100.00% 100.00% 100.00% 0.00% - - 12.317669 12.885468 4.450692 5.222909 7.956690 9.915553 1.526880 0.648268 0.0371 0.0312 0.0036 nan nan
2459883 not_connected 100.00% 100.00% 100.00% 0.00% - - 17.077965 17.917969 21.829704 23.866031 7.856229 10.840297 5.681985 3.276008 0.0388 0.0314 0.0047 nan nan
2459882 not_connected 100.00% 100.00% 100.00% 0.00% - - 28.426633 29.320604 23.203504 26.044085 11.125822 15.221884 2.306541 1.112326 0.0402 0.0320 0.0050 nan nan
2459881 not_connected 100.00% 100.00% 100.00% 0.00% - - 16.054580 16.725859 26.935571 29.872774 22.000013 30.505258 7.956442 13.655484 0.0379 0.0313 0.0040 nan nan
2459880 not_connected 100.00% 100.00% 100.00% 0.00% - - 19.536657 20.304672 22.869125 24.812129 6.747240 9.260832 2.772488 1.127504 0.0375 0.0316 0.0038 nan nan
2459879 not_connected 100.00% 100.00% 100.00% 0.00% - - 10.372082 10.896478 4.094860 4.787060 1.370322 2.015333 2.816725 1.551530 0.0374 0.0313 0.0035 nan nan
2459878 not_connected 100.00% 100.00% 100.00% 0.00% - - 17.598598 18.123279 27.750766 30.009272 11.515930 15.737673 7.264667 3.807254 0.0405 0.0321 0.0050 nan nan

Load antenna metric spectra and waterfalls from auto_metrics notebooks.¶

In [11]:
htmls_to_display = []
for am_html in auto_metric_htmls:
    html_to_display = ''
    # read html into a list of lines
    with open(am_html) as f:
        lines = f.readlines()
    
    # find section with this antenna's metric plots and add to html_to_display
    jd = [int(s) for s in re.split('_|\.', am_html) if s.isdigit()][-1]
    try:
        section_start_line = lines.index(f'<h2>Antenna {antenna}: {jd}</h2>\n')
    except ValueError:
        continue
    html_to_display += lines[section_start_line].replace(str(jd), f'<a href="{jd_to_auto_metrics_url(jd)}" target="_blank">{jd}</a>')
    for line in lines[section_start_line + 1:]:
        html_to_display += line
        if '<hr' in line:
            htmls_to_display.append(html_to_display)
            break

Figure 1: Antenna autocorrelation metric spectra and waterfalls.¶

These figures are reproduced from auto_metrics notebooks. For more info on the specific plots and metrics, see those notebooks (linked at the JD). The most recent 100 days (at most) are shown.

In [12]:
for i, html_to_display in enumerate(htmls_to_display):
    if i == 100:
        break
    display(HTML(html_to_display))

Antenna 113: 2459890

Ant Node A Priori Status Worst Metric Worst Modified Z-Score nn Shape Modified Z-Score ee Shape Modified Z-Score nn Power Modified Z-Score ee Power Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Discontinuties Modified Z-Score ee Temporal Discontinuties Modified Z-Score
113 N11 not_connected nn Shape 13.611979 13.611979 12.921914 5.612753 4.799823 9.069373 8.080866 -0.175549 0.906066

Antenna 113: 2459889

Ant Node A Priori Status Worst Metric Worst Modified Z-Score ee Shape Modified Z-Score nn Shape Modified Z-Score ee Power Modified Z-Score nn Power Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Discontinuties Modified Z-Score nn Temporal Discontinuties Modified Z-Score
113 N11 not_connected nn Shape 15.368499 14.375223 15.368499 3.558296 4.620022 10.966863 12.881762 3.052664 1.297265

Antenna 113: 2459888

Ant Node A Priori Status Worst Metric Worst Modified Z-Score nn Shape Modified Z-Score ee Shape Modified Z-Score nn Power Modified Z-Score ee Power Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Discontinuties Modified Z-Score ee Temporal Discontinuties Modified Z-Score
113 N11 not_connected nn Shape 12.815486 12.815486 12.042096 5.465487 4.643629 11.787880 9.848987 2.057200 2.392165

Antenna 113: 2459887

Ant Node A Priori Status Worst Metric Worst Modified Z-Score nn Shape Modified Z-Score ee Shape Modified Z-Score nn Power Modified Z-Score ee Power Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Discontinuties Modified Z-Score ee Temporal Discontinuties Modified Z-Score
113 N11 not_connected nn Shape 13.696026 13.696026 12.895705 5.371364 4.803014 10.296628 7.380940 0.754102 1.480593

Antenna 113: 2459886

Ant Node A Priori Status Worst Metric Worst Modified Z-Score ee Shape Modified Z-Score nn Shape Modified Z-Score ee Power Modified Z-Score nn Power Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Discontinuties Modified Z-Score nn Temporal Discontinuties Modified Z-Score
113 N11 not_connected nn Shape 18.618152 17.993098 18.618152 4.210142 4.813103 14.553657 14.978108 3.927069 3.236074

Antenna 113: 2459885

Ant Node A Priori Status Worst Metric Worst Modified Z-Score nn Shape Modified Z-Score ee Shape Modified Z-Score nn Power Modified Z-Score ee Power Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Discontinuties Modified Z-Score ee Temporal Discontinuties Modified Z-Score
113 N11 not_connected nn Power 26.014454 22.441739 21.950170 26.014454 23.551904 19.401089 14.145887 11.542472 16.917254

Antenna 113: 2459884

Ant Node A Priori Status Worst Metric Worst Modified Z-Score nn Shape Modified Z-Score ee Shape Modified Z-Score nn Power Modified Z-Score ee Power Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Discontinuties Modified Z-Score ee Temporal Discontinuties Modified Z-Score
113 N11 not_connected nn Shape 12.885468 12.885468 12.317669 5.222909 4.450692 9.915553 7.956690 0.648268 1.526880

Antenna 113: 2459883

Ant Node A Priori Status Worst Metric Worst Modified Z-Score nn Shape Modified Z-Score ee Shape Modified Z-Score nn Power Modified Z-Score ee Power Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Discontinuties Modified Z-Score ee Temporal Discontinuties Modified Z-Score
113 N11 not_connected nn Power 23.866031 17.917969 17.077965 23.866031 21.829704 10.840297 7.856229 3.276008 5.681985

Antenna 113: 2459882

Ant Node A Priori Status Worst Metric Worst Modified Z-Score nn Shape Modified Z-Score ee Shape Modified Z-Score nn Power Modified Z-Score ee Power Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Discontinuties Modified Z-Score ee Temporal Discontinuties Modified Z-Score
113 N11 not_connected nn Shape 29.320604 29.320604 28.426633 26.044085 23.203504 15.221884 11.125822 1.112326 2.306541

Antenna 113: 2459881

Ant Node A Priori Status Worst Metric Worst Modified Z-Score nn Shape Modified Z-Score ee Shape Modified Z-Score nn Power Modified Z-Score ee Power Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Discontinuties Modified Z-Score ee Temporal Discontinuties Modified Z-Score
113 N11 not_connected nn Temporal Variability 30.505258 16.725859 16.054580 29.872774 26.935571 30.505258 22.000013 13.655484 7.956442

Antenna 113: 2459880

Ant Node A Priori Status Worst Metric Worst Modified Z-Score nn Shape Modified Z-Score ee Shape Modified Z-Score nn Power Modified Z-Score ee Power Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Discontinuties Modified Z-Score ee Temporal Discontinuties Modified Z-Score
113 N11 not_connected nn Power 24.812129 20.304672 19.536657 24.812129 22.869125 9.260832 6.747240 1.127504 2.772488

Antenna 113: 2459879

Ant Node A Priori Status Worst Metric Worst Modified Z-Score nn Shape Modified Z-Score ee Shape Modified Z-Score nn Power Modified Z-Score ee Power Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Discontinuties Modified Z-Score ee Temporal Discontinuties Modified Z-Score
113 N11 not_connected nn Shape 10.896478 10.896478 10.372082 4.787060 4.094860 2.015333 1.370322 1.551530 2.816725

Antenna 113: 2459878

Ant Node A Priori Status Worst Metric Worst Modified Z-Score nn Shape Modified Z-Score ee Shape Modified Z-Score nn Power Modified Z-Score ee Power Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Discontinuties Modified Z-Score ee Temporal Discontinuties Modified Z-Score
113 N11 not_connected nn Power 30.009272 18.123279 17.598598 30.009272 27.750766 15.737673 11.515930 3.807254 7.264667

In [ ]: